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ABSTRACT 

This paper describes a system of built-in, or 
internal, evaluation used within the Botswana National Literacy 
Program ( NLP ) . Launched in 1981 and targeted toward illiterate adults 
and youth aged 10 years and older, the program aims at eradicating 
illiteracy in Botswana by 1986. The built-in evaluation was 
implemented in 1983, using program officials trained in a series of 
workshops as "barefoot evaluators." The practical realities of a 
Third World country with minimal infrastructure and scarcity of 
skilled manpower made classical methodological control impossible. 
The evaluation was therefore conceptualized as an experience creating 
& new culture of action within the NLP rather than as a technical 
experiment. Instruments served as frames for meanings rather than as 
yardsticks for measurement. Evaluation results have been gratifying: 
NLP program participation increased by 60 percent. Most collected 
data were usable and provided a coherent and credible program 
picture. Results could be utilized before formal report release. The 
experience of constructing this built-in evaluation sy3tem reinforces 
the view that such humanized methods are much more useful in Third 
World countries than external evaluations in the sophisticated 
research mode. (Author/BS) 
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Building a Built-in Evaluation System: 
A Case in Point 

ABSTRACT 

A system of built-in evaluation was designed and installed within 
the National Literacy Program of the Government of Botswana that seeks to 
bring about universal literacy in the country by the year 1986. 

The context demanded that local program officials be trained 
as evaluators and that the roles of local evaluators be conceptualized 
not as high level technicians but as "barefoot evaluators. 11 Within the 
practical realities f a Third World country, with minimal infrastructures 
and scarcity of skilled manpower, the design and installation of the 
built-in evaluation system was judged using criteria in the satisficing 
mode. Methodological control in the classical sense was impossible. 
Instruments sueh as questionnaires and observation schedules served not 
as yardsticks of measurement but as frames for meanings. 

Results of the first wave of program evaluation have been most 
gratifying. An evaluation system has indeed been built into the Botswana 
National Literacy Program and given the chance is likely to survive. 
All the District Adult Education Officers in the country have^en trained 
as also many of the Literacy Assistants in charge of clusters of learner 
groups in the districts. The evaluation exercise functioned concurrently 
as an agent of mobilization and increased number of participants in the 
program by $0%. 

» 

» 
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Most of the data collected was actually usable, and upon analysis 
provided a picture of the program that was both coherent and credible. 
Most Importantly, the results obtained from the evaluation exercise 
have already been utilized at various levels of decision making even 
before a formal report has been released. 

The experience of building a built-in evaluation system in the 
Botswana National Literacy Program reinforces the view that internal 
built-in evaluations in the satisf Icing mode are much more useful that 
external, stand-alone — and stand-alien — evaluations in the sophisticated 
research mode. More importantly, the experience of building built-in 
evaluations is transferable elsewhere in the Third World. It is within 
reach almost everywhere! 
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BUILDING A BUILT-IN EVALUATION SYSTEM 1 : 
A CASE IN POINT 2 



An understanding is slowly but surely emerging that log i cal -pos i t i vt st 
methodologies that have met great success in the physical and natural 
sciences are not always appropriate to human sciences; and that special 
approaches are needed for the study of human phenomena wherein facts 
and values are hard to separate (Polkinghorne, 19B3:x). 

Experience with program evaluations during the last two decades 
has led the best and the brightest of today's evaluation theorists and 
researchers to move away from classical models rooted in the "context 
of command 11 and toward new models grounded in the "context of accommodation 11 
(Cronbach, 1 9b0) which is the typical environment of education and 
development extension. A postposi tivi st era in program evaluation has 
certainly been inaugurated as fervent calls are made in behalf of 
responsive evaluation (Stake, 1975) > illuminative evaluation (Parlett 
and Dearden, 1978) > evaluation as connoisseurshi p (Eisner, 1979) • 
participatory evaluation (Hal I , 1978) » ut i I izat ion- focus sed evaluation 
(Patton, 197H)> evaluation in the natural i sti c mode (Guba and Lincoln, 1981), 
and evaluation as history (Cronbach, 1980). 

The hua,dnizatlon of evaluation theory and methodology has also 
led to evaluations interna 1 to program systems and to self-evaluations 
by non-experts, seeking to promote praxis through reflection on action 
looking for plausible causes rather than culprits (Bhola, 1979)* There is 

also an attempt to cover in program evaluations not only the interests 
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of donors and program administrators but *lso of other stakeholders 
thereby valuing program processes and impacts from multiple value 
pos i t ions . 

Program evaluations by external agents continue to be demanded 
by outside funding agencies to meet their own special needs of program 
planning and their politics of resource allocation. It is not always 
realized that while external evaluations end up serving no more than 
external needs, internal evaluations can serve both internal and external 
needs of decision makers. There is no reason why reports of internal 
evaluations could not be trusted and used by funding agencies. Such 
reports can be supplemented by additional data where necessary. On the 
other hand, external evaluations as stand-alone evaluations (Cronbach, 1980) 
can seldom serve the purposes of internal evaluations. Program functionaries 
and program participants experience external evaluations as spectators. 
The utilization of results obtained by stand-alone external evaluations 
by program planners and implementers has been found to be most problematic 
\Struening and Brewer, 1983). 

Claims of objectivity, validity and reliability made in behalf 
of external evaluations because these are conducted by external agents 
are often unwarranted. External agents may simply introduce one set of 
biases in place of another. They may get the numbers and miss the meanings. 
Reliability may be no more than a fabrication. In the technical assistance 
context, external evaluations are plagued by further problems. External 
evaluations are typically conducted by teams of evaluators sent by a donor 
agency from outside. Such teams of expatriates, almost as a rule, are 
unable to spend more. than a few weeks in the host country. Almost always 
they have to depend upon the locals as informers, interpreters and 
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investigators. T h e world they come to study is not available to them 
directly. Linguistic and cultural filters are placed between them and 
their respondents. This has led Campbell (1979:71) to suggest that 
external evaluations may be lacking in "the essential qualitative 
knowledge of what happended. 11 

THE CASE: THE BOTSWANA NATIONAL LITERACY PROGRAM 
Botswana (formerly, the British Protectorate of Bechuanaland) 
became independent in 1966. It is a country of the size of France, but 
two-third of it is covered by the Kalahari desert sand. Bulk of the 
population of around one million lives along the line of rail running 
North and South along the Eastern border. The rest is spread all over 
the desert in small habitations. 

Botswana is surrounded by five countries: Angola, Namibia, Zambia, 
Zimbabwe, and South Africa. Its politics is democratic, but its economy 
is greatly dependent upon South Africa. 

Most important development goals of the country have been defined 
by the government as employment creation and rural development (Republic 
of Botswana, 1980). Both require heavy inputs of education and extension 
for implementation. The government has realized that nonformal education, 
and especially adult literacy, will have to play a significant role in 
the development process. Some education must be made available to those 
who have been or continue to be bypassed by the formal system of education. 
At the same time, print communication seems necessary in a country where 
face-to-face delivery of extension services may be impossible 
to people living in small villages and cattle-posts spread all over the 
country. Radio can not alone carry the whole communication burden. 
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After some pilot testing in 1 979 and the experimental year of 1980, 
a National Literacy Program (NLP) was launched in Jun* to cover 

all the nine districts of the country. All illiterat its and youth 

ten years and older were targeted by the NLP. They were estimated to be 
some 20* of the relevant population, between 250,000 to 300, OuO in number. 
Illiteracy was to be completely eradicated by 19{J5/db. 

The NLP, after Its inception in 1981, attracted quite a bit of 
international attention and some international funding, especially from 
the Swedish International Development Agency (SIDAJ aftd the Agency for 
Technical Cooperation of the Federal Republic of Germany (GTZ) . A mid-term 
evaluation of the NLP had been agreed upon to be conducted during igb3 . 
It was in this context that the project of building a built-in evaluation 
system described here was implemented. The training inputs necessary 
for the implementation of the project came from the German Foundation 
for International Development, Federal Republic of Germany IDSE) . 

CONCEPTUALIZATION OF THE EVALUATION EXERCISE 
An internal evaluation of the Botswana National Literacy Program 
had been anticipated in a meeting of the National Literacy Committee in 
1981 before the NLP had received any international funding. 3 rhe concept 
of internal, built-in evaluation was elaborated further as part of the 
project described here. 

Internal evaluation was not to be a matter merely of substituting 
an evaluation team sent by a donor agency with a team of expatriate 
evaluators assembled by the Government. 14 Again, internal evaluation 
was seen as more than an in-house evaluation conducted by full-time 
evaluators on the regular establishment of the government and not responsible r °r 
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program planning and program implementation decisions. Internal evaluation 
was defined as an evaluation of the program by the program people themselves. 
It would combine the implementation and evaluation roles in the same one 
person . 

In attempting to build a built-in evaluation, the purpose was to 
make the total program system conscious of outputs and outcomes obtained 
or not obtained; sensitive to the need for program information to oe able 
to review actions and to design alternatives; to build local capacity 
for obtaining the required Information as and when needed; and to establish 
patterns of data collection, storage, retrieval and utilization that may 
become institutionalized in the near future. The program system was 
thereby to be put on a higher level of professional performance. 

The preceding set of objectives required that a particular level 
of evaluation capacity be built within the system through traini ng . 
Officials at the district level, the District Adult Education Officers 
(DAEO's), were targeted for such training. These DAEO's would be trained 
in the techniques of evaluation planning, design and implementation in 
a series of short evaluation workshops. They would, in turn, provide 
the necessary orientation to the Literacy Assistants (LA's) in charge 
of various clusters in each district. 

From a review of the qualifications and work experiences of DAEO's 
as a group (two or three had college degrees , and a few had diplomas in 
adult education from abroad), it was clear that the DAEO's could receive 
and profit from training in evaluation proposed to be provided to them. They 
would become more than M barefoot evaluators" — they could indeed be fitted 
with good enough sandles which would take them a long way! 

The overall evaluation exercise would not be conducted as an 
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experiment (there was nothing to experiment with, to compare under control), 
but instead would be conceptualized as an exper ience . It would not be 
a stand-alone event, but would touch the working lives of almost everyone 
within the NLP, thereby creating a new culture of action. This meant 
that all the fifteen DAEO's and «'l of the 120 or more LA's in the field 
at that time would directly take part in the evaluation exercise* As 
many of the 3>000 or more Literacy Group Leaders (LGL's) as possible would 
be associated with the conduct of the exercise. (Every one of them later 
would participate in the development of the Management Information System.) 
The evaluation exercise was to be an experience also for the participants 
in the NLP and as many of the learners, dropouts, graduates and local 
leaders would be covered as was possible within the eight to ten weeks 
of data collection. 

An important decision was made about data flow from the field to 
the center in Gaborone. The principle enunciated was; Use before 

Dispatch! One who collects, collates or receives data must first use 
it to improve the program at that level before releasing it to the level 
above. The LGL's were, therefore, asked to collect and collate data at 
the level of the learning center, in tables especially provided, and learn 
about their centers themselves before sending data up to the LA's at the 
clu c ter level . The LA's would col late all data at the cl jster level , in 
tables specially designed for the purpose, and write cluster reports on 
the basis of available data. They would, to the extent of their jurisdiction, 
act upon what they had learned from their reports. Then, they would send 
their reports along with supporting data to the DAEO's. The DAEO's would 
collate all data district-wise and write district reports. They too would 
act on their own reports, to the extent of their responsibilities, and 
then send their reports to the HQ in Gaborone, There a national report will 
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be written and used In decision making at that level and in negotiations 
with other government departments and outside funding agencies. 

THE METHODOLOGICAL MODEL OF EVALUATION 

The methodological model of evaluation used can best be described 
as di alect i cal . It was a dialectic between the ideal and the possible, 
between the standard and the satisf icing, between technical rationality 
and social processes — a marriage between the compass and the oar. 
The evaluation was unabashedly mul tlschematic and mul ttparadi gmat ic 
(Polkinghorne, 1 983 : xi ) , synthesizing contradictions as they appeared. 
Norms were projected, but approximations were accepted as long as the 
process kept on moving and participants did not get stuck (Schon, 1983)* 
The sole methodological objective was to describe, as best as possible 
in the context, the scope and the meaning of the NLP as it was being 
implemented. 

A rather formal matrix was imposed on the evaluation exercise: 
the NLP was analyzed as a system with various constituent subsystems; 
information needs of various subsystems were listed; outlines of a 
Management Information System were hammered out; an evaluation agenda 
relevant to the NLP at that point in time was agreed upon; evaluation 
studies were designed in terms of the agenda; , tools and 

instruments needed for data collection were prepared, translated and 
where possible pre-tested; field work was done ; and data obtained was 
collated and computer-analyzed. 

Within this formal matrix, there had to be tolerance for the 
approximation. Evaluation was being conducted, not within the context 
of control and command, but in the context of accommodation. The formal 
matrix had to have elasticity. The reality of lack of control, congruence 
and communication had to be faced. 
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The training Inputs necessary for building a built-in evaluation 
system had to be in English. While these training inputs had been 
designed with meticulous care, the usefulness of this training varied 
from one DAEO to another, depending upon their mastery of the language* 
There was also the constrain of training time. Decisions in regard to 
the information needs of the system, evaluation agendas and the elements 
of an MIS had to be made by the DAEO's and accepted even If they seemed 
less than elegant and comprehensive to the outside trainers. 

All tools and Instruments had to be designed by the DAEO's 
Individually or working In small groups. Faculty resources at the 
training workshops were adequate, yet there was a limit to how much 
quality control could be imposed and what kinds of revisions could be 
made. DAEO's could not always translate the instruments they had 
constructed, nor did they always agree on the translations made by others. 
Setswana is a new subject in the school system, introduced in the post- 
Independence period and good translators are hard to find. Checks on 
the quality of translation by retrans 1 at i on was not always possible 
and pre-testing was also not always done for lack of technical resources 
and constrains of time. 

The training of LA's as investigators had to be left to the 
DAEO's and was not always done systematically or well* Dependable data 
on the scope of the current program was not available for a sampling 
frame to be devised, and only general decisions could be made about 
who should be interviewed and where. 

Data collection could not be supervised in an evaluation that was 
conceptualised as an experience for the whole system. Some questions 
in the questionnaires and instruments did not communicate and the 
investigators ended up using these instruments net as yardsticks , but 
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as frames of reference for conducting conversations with their 
respondents. Data was recorded In Setswana and English and In some 
cases was a translated from a language other than Setswana. Once 
available, this data had to be coded, again, back to English using 
themes and key words from the English language. 

In a world of scarce resources of paper, printing, communication 
and transportation, there had to be a reflective conversation with the 
conditions (Schon, 1983) • Things were stretched to the limit, without 
snapping them. 

TRAINING IN ACTION 

The various stages of the process of building a built-in 
evaluation system, involving both training and action, are presented 
be low: 

Evaluation Workshop I (November 15_->o». 1982) 

Workshop 1 was conceived basically as a training workshop. 
It had two main objectives; (I) to learn the process of evaluation 
planning by actually doing evaluation planning; and (2) to write 
proposals, one each by each of the participating DAEO's, for conducting 
evaluation studies on topics relevant to the program needs of various 
districts, complete with evaluation objectives, evaluation designs, 
and tools and instruments; and to learn the basic techniques of evaluation 
in the process of doing so. 

Evaluation planning was taught using the systems approach.*' 

The "evaluand 11 , the NLP, was analyzed as a system of overlapping 
subsystems: (i) policy and planning subsystem ; (ii) administrative 
and instructional delivery subsystem; (iii) technical support subsystem; 
(iv) social mobilization subsystem; (v) curriculum and materials development 
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subsystem; (vi) taaching learning subsystem; (vll) post-literacy 
subsystem; (vIM) training subsystem; and (ix) evaluation subsystem 
(Bhola, 1984). An Ideal-type catalog of information needs of the 
various subsystems was developed and made the basis for developing a 
situation-specific evaluation agenda for the This agenda was 

validated through the social process of developing a consensus among 
the DAEO's and the HQ staff. It could not be taken to any other 
constituency at any other level. 

A distinction was made between the need for an MIS wherein 
program data routinely generated by the program could be stored for 
retrieval and utilization as and when appropriate; and evaluation 
studies conducted specially for meeting unanticipated information needs 
of the program. The new administrative patterns and role definitions 
necessitated by the built-in evaluation system were discussed as 
wel 1 . 

In the second phase of Workshop I, participating DAEO's** we re 
asked to select an information need they had already experienced in 
their work in the district and to design an evaluation study that would 
generate the information required. They were assisted in the design of 
their evaluation studies, in selection of indicators, sampling procedures 
and in the preparation of tools and instruments. Plans were made for 
translation and pre-testing of instruments and for field work, before 
the DAEO's returned to work in their districts. They would come 
back to attend another workshop some 15 weeks later where they would 
bring their data for collation and report writing. 

Evaluation Workshop II (March 7~18, 1983) 

The data brought to Workshop II by the returning DAEO's was 

14 
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uneven both in quality and quantity* Some of the DAEO's, back in their 
districts, had been unable to command the resources for typing, paper, stencils 
and ink, printing, personnel and transportation needed for the field work 
phase* Almost all of the DAEO's had had to lower earlier aspirations about 
the amount of data they would collect, but they all came to Workshop II 
with some data In hand. 

More importantly, they had all gone through their initiation as 
evaluators. They had all finalized their instruments in some form, had 
translated these instruments as best as they could and had administered them 
to samples of respondents. They had realized that respondents were not always 
accessible. They were frustrated by the realization that their questions were 
not always understood by their respondents and that they sometimes invited 
answers that were inappropriate or ununderstandable. They also understood 
the need of training their LA's if they were to be employed as investigators — 
something they did not follow through during the main evaluation exercise. 

The trainers had known from their earlier experience of conducting 
evaluation training in similar settings elsewhere in Africa?, that the 
processing and display of data for exploring relationships and patterns was 
never an easy task. Each of the DAEO's was, therefore, given a set of 
summary tables which could be used to answer the evaluation questions that had been 
originally posed. The DAEO's were then helped, individually, in p ocessing 
data they had brought to fit into the summary tables* Simple procedures of 
descriptive statistics were explained to those who needed to use such statistics. 
Workshop participants were also given guidance in report writing* 

These studies were based on rather limited data collected from a single 
district, and sometimes from a single cluster in a district, yet the insights 
these studies provided were highly significant. The studies on dropouts, on 
reasons for regular and irregular attendance, learner needs and special 
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problems of non-Setswana speakers in literacy classes taught in Setswana 
were more or less replicated later as part of the national evaluation 
exercise and did Indeed anticipate many of the findings of the national 
study. 

Judged as "barefoot eval uators", the DAFO's had done a good job 
and the training strategy being used had survived. The participants were 
asked to go home and complete their individual evaluation reports and 
bring them for presentation to Workshop III later in May 1 983 . 
Evaluation Work shop III (May 9-20, 1 983) 

Workshop III was designed to provide the transition from what had 
been essentially a training phase to what would primarily be an implementation 
phase of the built-in evaluation system. 

To provide a closure to the training phase, the workshop received 
and discussed reports of the evaluation studies already completed by the 
DAEO's. What had so far been individual experiences now became part of 
the group experience. Most of the findings fell within the range of 
expectations of the DAEO's and others working on the program and thus 
reinforced already existing knowledge. Soma findings, however, went against 
conventional wisdom and were hotly contested; and the studies that had 
produced those findings were challenged by workshop participants on the 
basis of methodology used, selection of samples and design of instruments. 

The workshop, for most part, was focussed on the core objectives 
of (i) recollecting the catalog of information needs of the NLP as first 
discussed during Workshop I; (ii) selecting, particioatively as a group, 

those information needs that must be given priority in the life of the NLP 
at its mid-point; (iii) agreeing upon the data that must become part of 
an MIS and upon the formats for data collection and storage; (iv) listing 
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urgent Information needs that must be fulfilled through specially designed 
evaluation studies; and (v) designing such evaluation studies, complete with 
tools and instruments for use at the national level. 

There were moments when as workshop director, the author wondered 
if we will come through it all. But the level of commitment of participants 
and workshop faculty was nothing short of amazing and the productivity of 
the workshop simply unbelievable. There was room for improvement in the 
tables designed for the MIS and many of the instruments designed for the 
evaluation studies could have profited from additional inputs of time 
and technique. But time was burning! Good enough was good enough. The 
barefoot evaluators would walk — though not always in great comfort. 

A Vi sit to the Field (August 1 5-26, 1 983) 

To ensure that things were happening as planned, the author made 
a field visit some 12 weeks after Workshop III. 

All was not well on the ground! Getting satisfactory Setswana 
translations of tools and instruments had not been easy. Plans for the 
pre-testing of instruments fell through when the University of Botswana 
lecturer who had agreed to supervise the pre-testing left the country. 
The printing of instruments involving close to a 100,000 sheets of 
paper in all — had strained the system to the limit. These instruments 
were waiting to be delivered to the various districts by truck which would 
take the driver more than two weeks on the road to make the full round. 

Down at the level of districts, things were no better. The DAEO's 
had not always provided the necessary orientation to their LA's to prepare 
them for their role in the evaluation exercise. They had sometimes told 
them what to do, but not how to do it. Plans for field work were no . 
where to be found. 
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Indeed, the LA's were often anxious and sometimes even hostile 
to the idea of the evaluation exercise. Some of them suggested that what 
the evaluation exercise what trying to find was already known and that 
in fact there was no scope for improving learner participation in the 
program in the midst of a drought when people were starving and had left 
their villages in search of food. 

In response to the existing situation, the field visit got 
transformed Into a series of 1-day workshops for LA's and their DAEO's 
in each of their district headquarters in the country, excepting two 
which were covered later. Using the data recently available from the 
national population census, the LA's were shown distributions of illiterates 
in their clusters and encouraged to recruit them into the NLP and serve 
their needs. The instruments to be used in the evaluation exercise were 
distributed to the LA's. The intentions of the various items in those 
instruments were discussed and some basic procedures of data collection 
were presented. The sampling methods to be used in data collection 
were also discussed. 

At the end of the field visit, it was not clear how the DAEO's 
and LA's will respond to the challenge of evaluation now before them.** 
The situation seemed hopeful and hopeless by turns. To introduce some 
certainty into the situation a "fail-safe" strategy was devised. The 
HQ staff in Gaborone, under the leadership of the Literacy Coordinator, 
would conduct an evaluat ion-wi thin-an-evaluation. As the national 
evaluation exercise was going on, they would work with a small national 
sample controlled from the HQ in Gaborone. It would be assured that 
data on this small sample would be collected and become available for 
analysis. If all else failed, they would have at least a stand-alone 
evaluation to fall oack upon. 

18 
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The Effort Bore Fruit 

Following the field visit by the author, the Literacy Coordinator 
and his colleague at the HQ, in Gaborone were out in the field most of the 
time during the data collection phase, going from district to district 
and from cluster to cluster. This may have turned things around. The 
field visit earlier during August 15"26, 1 983 may have shaken things up 
a bit as we 11 . 

Whatever the causes, the consequences were most gratifying. During 
the 10 to 12 weeks period before mid-December I983, 17,000 instruments had 
been completed and delivered: 7,000 dealing with learning needs of 
participants; 2,000 dealing with the special needs of non-Setswana speakers; 
1,600 on motivations of regular attenders; 1,150 dealing with those who 
had dropped out of the program; 3*000 listing uses to which literacy 
had been put by the new literates; and 1,500 village profiles describing 
what resources were available at the community level that could be used 
in providing services to the rural people. 

Samples were in flux. Various items in the evaluation instruments 
were found ambiguous and therefore unusable. The instruments in the hands 
of many LGL's and LA's became frames of reference for conversations with 
respondents rather than instruments in the classical sense. Yet, there 
was lot of countinq for the delineation of the scope of the program 
and lot of questioning in search of the meanings of the program. The 
evaluation exercise also became an instrument of mobilization for recruiting 
participants to the NLP. 

Seeing the big surge of data rolling in, the HQ in Gaborone decided 
to buy a microcomputer for electronic data processing. To beat deadlines 
for report writing imposed by higher level decision makers, data was 
collected from the DAE0's,and sometimes from LA's, before they hai had 
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the chance to process data at their levels and to write their reports 
as originally planned. Thus, the initial plans for data flow and 
information utilization were abandoned. The conceptualization of 
evaluation as experience was compromised somewhat because the product 
of evaluation (the report) was separated from the process of evaluation. 
Evaluation Workshop tV (De cember 11-17, 1983) 

The December 1 983 workshop had initially been planned as a workshop 
for finalizing district level reports and for sharing the findings. The DAEO's 
were to write their reports using the same one standardized format to 
enable comparison and cumulation of results for the purposes of a national 
report. Now that there was to be electronic processing of data, and 
most data was already at the HQ in the process of being coded, the program 
of the workshop had to be changed to fit new realities. The DAEO's 
spent their time developing "dummies 1 for their district reports, completing 
portions dealing with the ecological, economic, social and developmental 
context of their districts as well as in recording the human aspect of the 
evaluation exercise as they experienced it. 

The workshop also provided an opportunity for an evaluation of the 
to total project from November 1982 to December 1 983 in both of the aspects 
of training and action. 

T he Aftermath 

An outside technician was brought in from abroad to help in 
the electronic data processing. Data was coded with the help of students 
from the University of Botswana. It may be recalled, that the evaluation 
instruments had not initially been designed for computer analysis. Coding 
schemes had to be imposed on the data after the fact. This was not a 
simple task. The task was further complicated by the fact that the 
foreign technician was denied direct access to data because of the language 
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barrier. He had to uepend on local coders and judge the quality of data 
indirectly from the questions these coders happened to raise. 

A final report has been written. Since foreign donors have tied 
possibilities of further assistance with the findings of the evaluation, 
the government is understandably cautious. The report is going through 
a process of approval before being released. 

RESULTS OF THE EVALUATION EXERCISE 
Looking back at the project from the perspective of some eight 
months (at the time of writing), the following questions should be asked: 
Did the internal built-in evaluation project as planned and implemented 
achieve its objectives? Are the results of the evaluation study trustworthy? 
Is the built-in evaluation system as established likely to last and 
become institutionalized? Are the results obtained worth the time and 
effort put into the project? What were the incidental returns on the 
project, if any? 

An Evaluation was Completed . On the basis of an analysis of 
project documentation and informal interviews with DAEO's and LA's^, it 
is possible to assert that the evaluation was completed and was successful 
in regard to tha most important criteria of a f 1 : The current understanding 
of those working with the Botswana NLP about the program is an "improvement 
over previous understanding" (Pol ki nghorne , 1983:3). Both the scope and 
meaning of the program are clearly understood. 

As already indicated, some 17,000 questionnaires and interview 
schedules were completed as part of the evaluation exercise. The evaluation 
counted 28,800 learners enrolled in the program, ranging in age from 6 
to 9^, 3U maies and 63% females. Some 65,000 learners had been covered 
during the life of the program whicn has graduated some 20,000 literates. 
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This data may not satisfy the classicist interested in inferential 
uses of data but has provided a goldmine of data for suggestive uses in 
developing and modifying the program both in the long and the short run. 
While there may be a few discordant notes here and there, the overall 
coherence of the data is not Jeopardized, It makes lot of sense. 

The Meaning of the Program , Beyond numbers, there was considerable 
"qual i tative knowing 11 as was evidenced in the cluster reports written by 
LA's and informal reporting by DAEO's at the December 1 983 workshop. 
Program officials at the various levels of the program discovered that 
motivations were not spontaneous ,and that mobilization is necessary to 
promote recruitment. They developed a real feeling for the disadvantages 
of illiteracy and how literacy might be changing patterns of leadership 
i n 1 oca 1 commun i 1 1 es . 

They discovered that some of those they had been labeling as 
dropouts were really completers. These learners had left because the 
learning centers did not offer them much anymore. On the other hand > 
some learners did not want to leave. These centers were fulfilling their 
special social needs. It was discovered that literacy classes in Botswana 
do not just last 12 weeks during the months of September to December 
but that sonie of these are groups that last the year round. 

The evaluation also discovered primary schools within literacy classes 
In remote areas of Botswana where primary education had not reached, the 
literacy class acquired a dual meaning: it was both a literacy class and 
a pr imary school . 

Since the data and these many Insights had personalized meanings 
for the LGL's, LA's and DAEO's, the evaluation results were utilized in 
making program decisions even before any formal report came out. 
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Are the Results Trustworthy? As has been indicated earlier, the 
criteria of objectivity, validity and reliability in the classical sense 
do not apply. The criteria of coherence, credibility, conf i rmab i 1 i ty , 
fittingness and auditability do (Cronbach, 1980; Guba and Lincoln, 1981). 
The findings of the evaluation study meet both the truth test and the 
utility test (Weiss and Bucuvalas, I98O). The data collected and the 
insights garnered are In conformity with expectations of scholars and 
practitioners of literacy. Thus the truth test is met. The results also 
meet the utility test because they are action oriented, did challenge 
the status quo and have already moved the program in a positive direction. 

The Government of Botswana, Ministry of Education newsletter, 
Thuto , in a recent issue concl uded : n We ought perhaps to mention that 
the evaluation has been a very valuable source of information. As a 
result, we have much more information about the workings of the programme 
and if it becomes necessary to change it in certain ways, we can confidently 
take decisions that are in line with the findings of the evaluation.... 
The programme is already being modified as suggested by the evaluation. 11 

Will the Built-in Evaluation System Last? Conditions for the 
institutionalization of the built-in evaluation system have been met. 
The Government is not only favorable but infact claims ownership of the 
idea of a built-in evaluation. The series of four evaluation workshops 
has provided training to all the DAEO's in charge of the districts. Many 
of the LA's have acquired useful evaluation experience. 

An MIS is already in place. While maintaining continuity with 
the information collection procedures previously in force, it has strengthened 
and rationalized the system to make it more comprehensive, less repetitive, 
and usable for decision making at all the various levels of the system. 
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The NLP system has also acquired experience in evaluation having 
completed a full cycle of evaluation planning, evaluation design, construction 
of Instruments, data collection and analysis. It can muster resources 
from within to conduct a similar cycle on a new set of evaluation questions. 
Outside help, if at all needed, will be minimal. 

An Informal Cost-Utility Analysis, Was the evaluation exercise 
worth the time and effort and other resources put Into Its Implementation? 
The answer has to be a resounding M Yes M . If the costs of training Inputs 
by DSE and the expenditures on electronic processing of data (which was 
helpful but not crucial to the evaluation exercise) ^ 

are separated, we 

have an evaluation conducted on a shoestring budget. The whole evaluation 
exercise was conducted within the normal budget of the Department of 
Monformal Education of the Government of Botswana. In view of the 
collective evaluation experience it provided to the department, there 
could not have been a better bargain around. 

Incidental Returns on the Evaluation Exercise . The incidental 
returns on the evaluation exercise have been many and more keep on emerging. 
First and foremost, the evaluation exercise led to an impressive social 
mobilization in behalf of the program. In many cases, the evaluation 
provided the first opportunity for DAEO's and * LA's for a personal 
encounter with learners in the villages and cattle-posts. As a result, 
participation went from 18,000 to some 28,800. Institutional mobilization 
also resulted. The NLP was able to establish networks with other government 
departments of education and extension for providing integrated services 
to adult learners at the field level. 

Planning and evaluation are two sides of the same coin. Evaluation 
planning is the other side of program planning. In learning to describe 
the literacy program as a system in design terms for the purposes of evaluation, 
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the DAEO's developed useful Insights in the planning process per se. 

The DAEO's, after the evaluation exercise, are writing their routine 
monthly and quarterly reports with new insights. The reports are no more 
merely descriptive but have become quite analytical. For the first time, 
analyses at the learning group level and the cluster level have appeared 
In these periodical reports. 

The morale of those working within the NLP has improved. Many 
LGL's, LA's and most DAEO's have owned the eval uation. exercise and make 
references to it both in conversations and writing. There is an awareness 
of the need for professional izat ion in adult literacy, planning and 
evaluation. Some LA's have joined diploma and certificate courses in 
nonformal education and adult literacy under the department of correspondence 
education of the University of Botswana. 

There is some fallout on other department of the Ministry of Education. 
The Correspondence Education Department of the Ministry is anxious to 
train their officers in evaluation in the same mode preparing them to apply 
those skills to course evaluation ano learner evaluation. There is also 
interest in curriculum evaluation and one of the recent workshops for 
teachers engaged in curriculum evaluation used the project handbook. 
Evaluating Development Training Programs (Bhola, 1982). 

Finally, there is now available a model of how to build built-in 
evaluation systems within literacy and other development programs in 
the Thi rd World . 

LESSONS FOR ELSEWHERE 
There are some useful lessons in the description of this project 
for planners and organizers of literacy campaigns, programs and projects 
elsewhere in the Third World: 
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1. Systems of Internal evaluation integrated with the process of 
program planning and implementation can Improve the effect! /eness of 
literacy campaigns, programs and projects. Such evaluations, on the 
one hand, professionalize the system, and,on the other hand ,mobi 1 ize 
both institutions and learner participation, 

2. The concept of "barefoot evaluator" is realistic ; and the system 
of built-in evaluation system as described above is replicable in most, 
If not all, Third World countries working on literacy campaigns, programs 
or projects. 

3' The Action Training Model — encompassing a series of workshops 
to provide training to program officials in evaluation techniques tied 
directly to the act of conducting an evaluation is workable and 
should be tested in other settings. 

k. Finally, the lesson must be learned that experience is a great 
teacher. Lessons learned from the above exf^rience of building a built-in 
evaluation system must now be put to work. More attention will have 
to be paid to design of instruments, their pre-testing and use. Field 
investigators will have to be better trained for data collection. There 
will have to be better field organization and better scheduling of data 
collection. 

Next time around we must tolerate only a different set of errors! 
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NOTES AND REFERENCES 

NOTES 

1. The choice and presentation of facts and the views expressed 
in this paper are the sole responsibility of the author and do not in 
any way represent or commit any government, institution or individual 
who may have collaborated with project here described in any way and 
at any stage of the projets's planning or implementation. 
2. The case In point is the Botswana National Literacy Program which 
is the responsibility of the Department of Nonformal Education, Ministry 
of Education, Republic of Botswana. 

The program is receiving assistance from a variety of international 
donors, among them, the German Agency for Technical Cooperation (GTZ) ,Eschborn , FRG. 
GTZ also sent a German expert to Botswana to work as the Literacy Coordinator 
for the National Literacy Program. 

During 1982-83, the author was retained by GTZ to provide backup 
technical support to the National Literacy Program (NLP) in various aspects 
of planning, training, evaluation and networking. Grateful thanks are due 
to Dr. Wolfgang KGper, Dr. Herbert Bergmann and Ms. Inge Eichner of GTZ 
for the opportunity so provided. 

The training inputs were made by the German Foundation for International 
Development (DSE) , Bonn, FRG. Thanks are due to Dr. Josef Miiller of DSE for 
inviting the author to direct the whole series of workshops and for the 
trust placed in the author. 

The built-in evaluation system described here was not built In a day 
nor was it built by any single person. The project described here covered 
a period of eighteen months during I982-83. Those directly and significantly 
involved in the design" and implementation of the project were: Dr. Josef 
Muller of DSE; Dr. Volkhard HundsdSrfer, GTZ-sponsored Literacy Coordinator in 
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the Department of Nonformal Education, Ministry of Education, Gaborone, 
Botswana; Dr. Ul la Kann, SIDA-sponsored Planning Officer in the Ministry 
of Education; Government of Botswana Officers in the Ministry of Education, 
and especially those In the Department of Nonformal Education; the 
District Adult Education Officers (DAEO's) posted in the districts; Literacy 
Assistants (LA's) in charge of clusters of learning centersjand Learning 
Group Leaders (LGL's) who work with adult learners in the field. 

In the conduct of the four training workshops for DAEO's, my 
esteemed colleague and dear friend, Dr. Josef Muller played a most significant 
part. He was solely responsible for the planning and all organizational 
details of the workshops. Though not so stated officially, he . co-designed, and 
co-directed all the four workshops with the author. 

The following agreed to serve as faculty members at the various 
workshops and gave generously of their time and expertise, often at 
considerable oersonal sacrifice: Dr. Lars Mahlck, International Institute 
for Educational Planning, Paris; Dr. Ash Hartwell, Unesco Education Planning 
Adviser, Maseru, Lesotho; Dr. Peter Higgs, Unesco Specialist in Literacy, 
Lilongwe, Malawi, Dr. Volkhard Hundsd6 r fer, Literacy Coordinator, Gaborone, 
Botswana; and Dr. Josef Muller of DSE in the November 1 982 workshop; 
Ms. Elvyn J. Dube, Institute of Adult Education, University of Botswana, 
Gaborone; Dr. Linda Ziegahn, USAID Researcher, Lesotho Distance Teaching 
Center, Maseru, Lesotho; Dr. Volkhard HundsdoVfer, Gaborone; and Dr. Josef 
Muller in the March I983 workshop; Ms. Elvyn Dube; Dr. Volkhard Hundsdorfer 
and Dr. Josef Muller in the May 1 983 workshop; and Dr. Vo'khard Hundsdorfer and 
Dr. Josef Muller in the December 1 983 workshop. 

Dr. Volkhard Hundsdorfer as Literacy Coordinator in the Department 
of Nonformal Educatiqn, Ministry of Education was responsible for all 
local arrangements in connection with the series of training workshops ; 
and was the officer in charge of the implementation of the evaluation 
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in all its aspects. 

All those whose work is acknowledged above . share the credit 
for whatever was achieved in this poineering effort of building a built-in 
evaluation system in a national-scale literacy program. I must, however, 
take all the blame for the failure to anticipate problems and for errors 
of judgment. 

3. Credit for this decision must go to Edwin Townsend-Coles , at 
that time the Chief Education Officer in charge of the newly founded 
Department of Nonformal Education and his colleagues in the Ministry. 

4. The internal built-in evaluation was planned and installed in the 
Botswana National Literacy Program by agents external to Botswana. The 
training of local officials in evaluation techniques was also handled 

by outside consultants under the aegis of DSE. The important point to 
remember , however , is that both the planning of the evaluation exercise 
and the training in evaluation techniques were organized part i cipati vely 
and all significant decisions involved were under the control of local 
officials. 

5* The systems approach used in evaluation planning, again, was by 

no means new to the project. Dr. Ulla Kann in her paper, "How Can We 

Succeed?:A Self-Evaluation of the National Literacy Programme in Botswana, 11 

presented to the Project Development Workshop under the International 

Institute for Educational Planning, Research and Training Project on Evaluation 

and Monitoring of Educational Reform Programs in Africa, Moshi , Tanzania, 

March 29 to April 6, 1982 had used the systems approach in delineating 

the evaluation needs of the Botswana National Literacy Program. 

6. Evaluation Workshop 1 (November 15-26, 1 982) was organized in 

collaboration with the International Institute for Educational Planning 

to serve concurrently the needs of the Institute's project: Research and 
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Training Project on Evaluation and Monitoring of Educational Reform Programs 
in Africa. Teams of four evaluators each came to join the workshop from 
Lesotho and Malawi. The March 1 983 workshop was also attended by teams 
from Lesotho and Malawi. The May l 983 workshop was attended by the Malawi 
team but not by the team from Lesotho. The December I983 workshop was 
attended only by the national staff of the Botswana National Literacy 
Program. 

1. The author In association with Dr. Josef Muller, and under the 
aegis of the DSE, during 1979-82, had organized a series of evaluation 
workshops for the training of trainers of development workers using an 
Action Training Model (ATM). See H.S. Bhola, Action Training Model (ATM) — 
An Innovative Approach to Training Literacy Workers. Notes and Comments #128. 
Unit for Co-Operation with UNICEF and WFP, Unesco, Paris. March I983. 

8. A mission report was written at the end of the field visit. This 
mission report was used to make some important decisions in anticipation 
of the results of the evaluation that would be completed only in December 
1 983 - This led to serious misunderstandings among various stakeholders. 

9. The author was in Botswana, again, during August 13-24, 1984; and 
had the opportunity of administering questionnaires dealing with the 
usefulness of the evaluation exercise and the effectiveness of training 
for the evaluation exercise. While the sample was small -- about half 
of the DAEO's in the country and no more than half a dozen LA's out 

of a possible 125 — there were some useful insights that became available. 

10. This is not to disount in any way the most competent contribution 

of Dr. Rainer Hampel who came from Germany to help the Literacy Coordinator 
in the electronic processing of data and gave so willingly of his time 
and effort. 
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11. This paper was presented to The 198*1 Joint Meeting of Evaluation 
Research Society and Evaluation Network held In San Francisco, California, 
October 11-13. 198**. The assistance of Joginder K. Bhola in the writing 
of this paper is gratefully acknowledged. 
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